Skip to content

OSS-Fuzz: Move fuzzer to upstream repository - #6203

Draft
arthurscchan wants to merge 3 commits into
pybind:masterfrom
arthurscchan:fuzz/upstream-harness
Draft

arthurscchan wants to merge 3 commits into
pybind:masterfrom
arthurscchan:fuzz/upstream-harness

Conversation

@arthurscchan

Copy link
Copy Markdown

This PR moves all the OSS-Fuzz fuzzers to the upstream storage for easier management. This PR works together with OSS-Fuzz (google/oss-fuzz#16274) to accommodate the moving of fuzzers upstream.

Of course, this is only a suggestion. If the maintainers would prefer the fuzzers to remain in OSS-Fuzz, I am happy to keep the current setup and to ping the relevant maintainers for review whenever I add new fuzzers later in the process.

Thank you for considering this, and I would welcome any feedback on the preferred approach.

Remark: After this PR is accepted and merged, there will be additional PRs for adding new fuzzers for this project later.

@arthurscchan
arthurscchan force-pushed the fuzz/upstream-harness branch from 20f42db to 26dc0e3 Compare October 13, 2026 09:07
Comment thread tests/test_gil_scoped.py Resolved

@rwgk rwgk left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for proposing this! Keeping the fuzzers alongside the code they exercise would let us review and update both together.

My concern is that this PR adds the source file without integrating it into our build or tests. Changes in either pybind11 or the OSS-Fuzz environment could then break the harness while all pybind11 CI checks remain green. Before we take ownership of this code, I’d like us to establish a sustainable way to build, exercise, and maintain it.

I don't have experience working with fuzzers, so the requests below were mostly written by codex GPT-6.1-Sol ultra, with small manual edits:

Could you please add the following?

  1. A supported build target for the harness. Ideally, our tests and OSS-Fuzz would use the same build rule. Making this optional and enabling it in one suitable CI configuration would be sufficient; we don’t need LLVM dependencies throughout our test matrix.

  2. A small checked-in corpus and an automated replay test. This should exercise the actual fuzz callback, with inputs covering both successful operations and expected Python exceptions. There should be at least one CI job.

  3. An automated check of the OSS-Fuzz integration. CIFuzz looks like a suitable option for building and briefly running the fuzzers on PRs. Another approach would also be welcome if it checks compatibility with the actual OSS-Fuzz build environment.

I’d also appreciate an explanation of which pybind11 behavior this harness is intended to exercise, particularly because it involves py::exec. When a failure is reported, we’ll need to reproduce it locally, understand the operations involved, and reduce it to a focused regression test. Could you document that workflow? Would fixed Python snippets with fuzzed arguments, or a selection of explicit binding and conversion operations, make failures easier to investigate? I’m open to your advice on the approach; I’d like us to establish that the harness provides useful coverage and that maintaining and debugging it will be practical for pybind11 contributors.

@arthurscchan

arthurscchan commented Oct 28, 2026 •

Copy link
Copy Markdown
Author

Thanks for your reply. I will address those concern and try implementing that into the build/test system, please spare me for a few days to handle that and sorry for the slightly late reply. @rwgk

@arthurscchan
arthurscchan marked this pull request as draft October 29, 2026 04:55
@arthurscchan
arthurscchan force-pushed the fuzz/upstream-harness branch from 52e7a4a to 8a6e88c Compare October 29, 2026 04:55
@arthurscchan

Copy link
Copy Markdown
Author

Hi, I have a push a drafted version that addressing your comments and suggested approach. Here are the summary and I am doing longer test to verify some ofthe changes, Please do let me know if you find additional comments and suggestions on the work, thanks. Remark: Part of the README and cifuzz process has been helped by AI.

  1. Dropped the test_gil_scoped.py change on rebase. That was a pre-commit side effect; test: fix GIL subprocess test correctness #6211 and test: consolidate GIL subprocess runs #6212 cover that file.
  2. Added an opt-in PYBIND11_BUILD_FUZZERS target that builds tests/fuzz/pybind_fuzzer.cc against pybind11::embed. It is off in the normal matrix. OSS-Fuzz uses the same target when LIB_FUZZING_ENGINE is set.
  3. Checked in a small corpus and one Ubuntu 3.13 CI job that replays it through LLVMFuzzerTestOneInput. The inputs cover successful Python and expected error_already_set cases.
  4. Added a CIFuzz workflow for the pybind11 OSS-Fuzz project. It is non-blocking until pybind11: Move fuzzer to upstream repository google/oss-fuzz#16274 points build.sh at this target, because CIFuzz uses that script rather than this CMake option.
  5. Documented the coverage and the reproduce/reduce workflow in tests/fuzz/README.md.

Signed-off-by: Arthur Chan <arthur.chan@adalogics.com>
@arthurscchan
arthurscchan force-pushed the fuzz/upstream-harness branch from 7bdc94d to 5f09526 Compare October 29, 2026 09:45

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants